fix(falkordb): bound standalone switchover action - #3265
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3265 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 163 165 +2
Lines 25203 25621 +418
======================================
- Misses 25203 25621 +418 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
leon-ape
left a comment
leon-ape
left a comment
There was a problem hiding this comment.
This changes the switchover timeout under the existing falkordb-4-1.2.0-alpha.0 ComponentDefinition identity, but existing Components are not reconciled from ComponentDefinition updates and kbagent reads the action timeout from its startup environment. An in-place addon upgrade can therefore leave existing FalkorDB pods on the old 30-second limit, so the bounded switchover still terminates before Sentinel convergence and targeted switchovers remain unavailable on upgraded clusters.
|
Addressed the existing-cluster timeout P1 at exact head The timeout contract now ships under new ComponentDefinition identity Independent focused re-review: |
leon-ape
left a comment
leon-ape
left a comment
There was a problem hiding this comment.
The shipped Upgrade OpsRequest migrates only the falkordb component, while the alpha.1 main ComponentDefinition resolves its Sentinel component, credentials, and pod roster through exact falkordb-sent-4-1.2.0-alpha.1 references. An existing replication cluster keeps its Sentinel sibling on alpha.0, so the recreated main Pods lose those optional Sentinel variables and start via the lexicographically first FalkorDB Pod instead of the authoritative Sentinel primary; after a prior failover this can promote a stale replica and replicate that stale history across the cluster, losing acknowledged writes. The verifier checks only the main Component and Pods, so it reports this mixed-definition topology as successful.
The migration and README treat an omitted serviceVersion as preserving the cluster's current version, but the public Upgrade API defines omission as selecting the latest compatible version; preservation is only the current controller's private nil behavior. A conforming controller change or a newly published compatible version can therefore move the database image during this timeout-only migration, and the verifier still succeeds because it never checks the resolved serviceVersion or image.
|
Closed the P0/P1 review findings in exact head
Static/source gates on this head's committed diff: focused verifier+contract ShellSpec 17/0; full FalkorDB ShellSpec Bash 5 246/0; Bash 3 59/0 with 4 declared version skips; Helm lint/render, production ShellCheck, Bash syntax, release-note parse, and No KubeBlocks runtime migration was executed as part of this source closure, so this does not claim runtime acceptance. |
3 participants
Root cause
A real standalone FalkorDB Switchover requested a specific replica, but kbagent stopped the lifecycle action at its 30s default before the script's 300s convergence check could finish. The live ComponentDefinition had
timeoutSeconds: 0, which selects that default.Simply disabling the kbagent timeout would make the action unbounded and could leave temporary replica priorities behind. Updating the same ComponentDefinition identity also would not regenerate existing Pods'
KB_AGENT_ACTIONstartup environment.The existing-cluster migration also needs to preserve two contracts explicitly: alpha.1 main Pods must continue resolving an alpha.0 Sentinel sibling, and a timeout-only upgrade must not let KubeBlocks select a different compatible service version or database image.
Changes
timeoutSeconds: -1so kbagent does not preempt Sentinel convergenceredis-clicall to 5s plus 2s kill gracereplica-priority=0on non-candidatestimeoutis unavailable, cleanup fails, or Sentinel never confirms the biasfalkordb-4-1.2.0-alpha.1falkordb-sent-4serviceVersionSucceed, exact target ComponentDefinition, complete old-Pod UID replacement, non-terminating Ready replacement Pods, and exactly one serialized switchover action withtimeoutSeconds: -1Verification
Exact head:
ebfde18310304ffd1c6044a5c7e373604abc5684VERDICT: NO BLOCKERgit diff --check: PASSRuntime focused validation remains owned by Test and must pin this exact commit. These source, mock, and CI results are not a topology runtime PASS.